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Status Report for Summer 1985 


Bruce D. Verner 
Intelligent Systems Laboratory 
Xerox Palo Alto Research Center 
26 August 1985 


This document is a brief summary and status report on my research at 1SL/PARC during 
the summer of 1985. It describes some of the research, software engineering, and other 
activities 1 performed for the User System Research (USR) group of ISL. For information 
on my work as a Research Intern at PARC during the previous summer, refer to the August 
1984 technical mini-report entitled "A Brief Summary of My Work on TROOPS." 

According to telephone discussions before my return to PARC, my primary project was to 
be the augmentation of the Input/Output facilities of the Dandetiger computer through 
electronic hardware and software extensions. Upon my arrival at PARC in early June, 

Austin Henderson informed me that the information-space flyer project is receiving higher 
priority and asked me to devote my efforts toward advancing that work. The 
information-space flyer concept is similar to a flight simulator where one traverses over^n ' 
information topology (such as a semantic network) rather than a geographical terrain./He 
explained (as Stuart Card had previously mentioned) that the Human Interface Program at 
the Microelectronics and Computer Technology Corporation (MCC) has a similar research 
project in progress and that we want to keep the lead in this area. In addition to advancing 
the state-of-the-art in user interfaces and information display, our pursuit of this 
hypermedia project is contributing to the formulation of a general comprehensive theory of 
visual interfaces. (Hypermedia are presentational media which perform in 
multidimensional ways. Some of our research is related to the pioneering ideas of Ted 
Nelson such as the Fantics and Thinkertoys 3-D concepts.) Stu and Austin had completed a 
considerable amount of work on the underpinnings for a net flyer environment for the 
Dandeiris 3-D color graphics system, but much remained to be done. 

Austin gave me a few pointers to the existing work which consisted of, the IRIS graphics 
software library for Interlisp-D; the Intermediate-Displayable Network (ID-Net) package 
for building, manipulating, and accessing object-oriented network data structures; the 
Organization Chart (Org-Chart) package for generating visual representations of ID-Nets 
on the Dandetiger; and the Whiteboard (White- Board) package for displaying the outputs 
of Org-Chart in Euclidean 3-space on the Silicon Graphics IRIS terminal. 

After discussing the desired future directions of this research, l proceeded to learn the 
details of the high-performance, pipelined, custom VLSI-based IRIS graphics processor -- a 
truly remarkable machine. I supplemented my previous knowledge of computer graphics 
theory and practice by reading Principles of Interactive Computer Graphics by Newman 
and Sproull (both formerly of PARC) and various other relevant works. Then, I read the 
IRIS documentation manuals to become intimately familiar with the functioning of that 
system. The hardware and software architectures of the IRIS seem to be heavily influenced 
by the ideas of Newman and Sproull, and thus, I recommend their book to anyone 
interested in the deeper and more formal concepts underlying this technology. (This 
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impression is not surprising when one considers that Silicon Graphics has several PARC 
alumni.) Once acquainted with the research vehicles, I began modifying and extending 
these systems in accordance with Stu's and Austin's requests and suggestions and my ideas. 

The first software package I wrote adds a facility that generates and displays function calling 
path trees on the IRIS in response to the issuance of a Masterscope "Show paths ..." 
command by an Interlisp-D user. This tool is called Master-Chart and is located in the file 
{Phylum}<Vemer>Lisp>Net>Master-Chart(.dcom). When Master-Chart is loaded, it taps 
into the Masterscope database by advising the appropriate functions in the Browser 
package. In response to a user's "Show all paths from aFunction" query, Master-Chart 
(when enabled by a flag) creates an ID-Network representing the calling paths. Each node 
in the net contains information about the corresponding function. The resultant network is 
then displayed on the Dandetiger screen in an Org-Chart browser window for manipulation 
by the user. With the browser, various extractions (information organizations) and layouts 
(extraction display formats) may be generated from the information net upon request. The 
user then selects a menu command to create and display a whiteboard world on the IRIS 
showing the desired net layout. A flyer may be used to explore the presented relationships 
between functions. The Master-Chart package, in conjunction with Org-Chart, allows the 
display of large and complex function interdependencies that exceed the viewing capacity 
of the existing tools developed for the D-machine screen. 

Having completed Master-Chart, I wanted a way to explore the network domain displayed 
on the IRIS that is easier than the former method of typing coordinate and velocity codes 
on the keyboard. 1 set out to design and implement a mouse-driven information space flyer 
for White- Board. What resulted is a mouse-controlled vehicle that models an inertial mass 
with thrusters in a frictional medium. The physical dynamics of this explorer are more 
similar to those of a submarine or heavy aircraft than a spaceship -- the inclusion of friction 
in the model serves to protect the flyer from wild and difficult to control movements that 
are a familiar annoyance to users of spacecraft simulators. The mouse controls for this flyer 
are described in the document, "Summary of Controls for Info-Net Flyer #1." 

Lennart Lovstrand has recently suggested and plans to implement what he calls an impulse 
flyer. 1 interpret this to be a sort of "hyper-warp" device that does not observe the physical 
law that the second derivative of the position of a mass must be a continuous function. If 
the velocity of the flyer is not "everywhere differentiable", then the vehicle is free to make 
discrete jumps in space or halt in zero time. These behaviors should be carefully controlled 
to avoid disorienting the user. Still, Lennart's impulse flyer seems an interesting idea and 
will be useful for comparison with mine. 

There are certainly many possible kinds of flyers for our information spaces, and we hope to 
extract what makes a good flyer from our series of experiments. Toward this end, Stu has 
suggested the separation of flyers from the rest of the White- Board system to facilitate the 
rapid interchange of prototype vehicles. In my opinion, this test-bed approach to the 
White- Board structure is worth pursuing. Perhaps the model for a modular flyer should be 
a state-machine which interfaces to White- Board through state variables refering to physical 
parameters such as position, velocity, acceleration, and coefficient of friction. Eventually, 
the user should have a hanger of craft from which to select according to desired 
performance properties. 
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With my completed flyer, I traversed the information space and discovered the need for 
several refinements to the existing White- Board display paradigms. One problem was that 
the field-of-view of the flyer could be rotated away from the board - rendering the network 
out of view. In this situation, one is left looking into empty space with no orientation clues 
and little chance of easily finding the whiteboard. To solve this problem, I recalled the 
relevant theorems of trigonometry and analytic geometry and derived constraint equations 
on the rotation of the flyer to keep the whiteboard in view. The code for this feature is 
included in Master-Chart and may be put into effect by setting a global flag. 

Another shortcoming is that the ” flashlight" (a dot which indicates a point of interest on the 
whiteboard tow-ard w'hich to gravitate) was of constant size relative to the board. Thus, at 
farther viewing distances from the board the flashlight would become vanishingly small -- 
leaving the user confused about the location of the current focus. To remedy this situation, 

1 modified the flashlight program code to make the beam diameter proportional to viewing 
distance. Now, the flashlight is approximately a constant size relative to the IRIS viewport 
and always readily visible without being obtrusive. It is interesting to observe that the 
completion of a first flyer precipitated the discovery of the need for these refinements to 
White-Board. 

All of the previously described work was then demonstrated to Stu and Austin for feedback, 
and they gave a positive response. 

We wanted to demonstrate our hypermedia system to a group of visiting scientists, but it 
was lacking a major capability -- that of printing text in IRIS displays. Such a feature is 
necessary for labeling the various network structures in the IRIS world space. Since the 
need for text printing was immediate, I designed and implemented an object font package 
which enables the generalized printing of text as graphical objects on the IRIS. During 
White-Board initialization, this code reads a file of font data (previously generated by 
Austin) and compiles that representation into a list of IRIS display library commands. 

When EVA Led, this expression builds an object font display-list data structure within the 
IRIS. A font table of characters, character codes, widths, kearns, and object numbers is 
created for later use by my printing mechanisms. The compiler-based approach is efficient: 
because the translation of source data to library commands need only be perfomed once. 
IRIS text display is then available through a transparent programmatic interface. 

Parameters such as height, position, color, linewidth, and format can be easily modified on a 
character basis. The Object-Font package was successfully completed and used to label 
Master-Chart nodes with corresponding function names, to print multiple text lables on 
Xerox organizaton charts, and to paint numbers on the cards in Lennart's arithmatic 
problem mockup. The commented code of the object font package is avaliable in the file 
{Phylum}<Verner>LISP>Net>Object-Font.(dcom). 

As requested by Austin, I added to White- Board the capability to display a network of the 
Xerox organization chart with scaled text labels on each node for post, title, and name. 

Three different colors are used lor the three label types allowing easy identification of 
information. When enabled by setting a flag, a labeled org-chart will be automatically 
generated from a text file representation. The default text file name is contained in a global 
variable which may be reset by the user. Refer to the documentation in Object-Font and 
Master-Chart for further details. 



4 


At one of the discussion sessions, Stu suggested that we try to adapt to our system the 
powerful "fish-eye view" display paradigm developed by George Furnas at Bell Labs. 
Furnas had described the advantages of this algorithm during his visit to PARC. 1 obtained 
a copy of Furnas' paper on the subject, "The FISH EYE view: a new look at structured 
files", which I subsequently read. Lennart offered to write a static fish-eye view as another 
Org-Chart layout option, while I volunteered to create a dynamic-incremental fish-eye view 
that would reside in White- Board itself. 

The existing structure of White- Board had all the arcs and nodes of the net grouped into 
one IRIS display object -- a configuration unsuitable for the implementation of dynamic 
fish-eye views. To facilitate the construction of such views, I rewrote and extended much of 
the White- Board code. One of the major changes was making the net constituents into 
separate IRIS display objects. Now, each node can be edited to change its scale, color, or 
other properties independently of other nodes. The node selection behavior in Org-Chart 
was changed to choose the closest node to a mouse click rather than reporting a miss, and 
the node selection facility was extended to work on IRIS whiteboards. This fish-eye view 
test-bed was exercised by adding code that displays the node nearest the flashlight in red at 
an expanded size. The flashlight is used to select a node on the whiteboard in a manner 
similar to the use of the mouse cursor in Org-Chart display windows. Text labels on the 
node are also expanded accordingly. When a new point of interest is chosen, the previously 
selected node is reverted to its former ordinary appearance and the desired node is 
highlighted. The documented code for this work is mostly contained in the file 
{Phylum }<Verner>LlSP>Net>Proto-Fish-Eye(.dcom), and other parts of it are in the - 
Master-Chart package. With this solid foundation completed and demonstrated, I was 
ready to attempt the creation of a full-fledged fish-eye view engine. 

Stu suggested that we attend the Siggraph conference in San Francisco to gather 
information on progress in the field of computer graphics. At that event, we observed many 
interesting developments; although much of the exibited technology was imitative and 
uninspired. Silicon Graphics (IRIS) and Lucasfilm (Blitter machine) where among the 
leaders in hardware. Abel Graphix, some of the smaller companies, and the universities 
(Caltech, Tokyo, ...) had the best software. The poor quality of the PARC/Apple-like 
windowing user interfaces that I tried on various machines was disappointing. Obviously, 
the developers of this hardware and software have ignored the well-refined paradigms 
stated in the Smalltalk-80 books and fashioned their systems in an inconsistent, inelegant, 
and ad-hoc manner. Yet this is still progress over the, previous conditions, and they will 
eventually correct their errors. After much manuevering, we managed to obtain tickets for 
the film and video show -- an enjoyable presentation that spotlighted some of the recent 
advances in the art and science. 

Around this time, we received from Silicon Graphics the new IRIS workstation upgrade for 
the laboratory. The IRIS workstation has various significant increases in capacity and 
capability over the old IRIS terminal, including the ability to run the UNIX environment. 
Our intention is to reimplement the time critical segments of the White- Board main control 
loop activities in the C language on the IRIS. The Dandetiger will then send to an XNS 
network daemon (process) on the IRIS any adjustments needed to keep information 
consistency. We expect this restructuring of our software architecture to yield significant 
performance gains. 
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After installing the workstation, we had severe difficulties with the multitasking mode of 
UNIX. I contacted Silicon Graphics, and with their assistance and Lennart's help isolated 
the problem to the Ethernet board in the IRIS. After much persistence, the problem was 
resolved by having SG replace the card. Lennart has configured user accounts for the 
workstation and we are becoming proficient with the new features of the system. 

The IRIS workstation requires the development of new communications and graphics 
library software in Interlisp-D. This task has been almost finished by Greg Nuynes at 
Xerox AIS; and he is continuing to debug the IRISlib package. I have sent him "bug 
reports" describing the details of several fatal problems in the code. He is proceeding to fix 
these and we await the completion of a fully functioning release so that we can begin use of 
the new IRIS. The work of Greg and Michel Desmarais (formerly at PARC) on the systems 
related programming has been essential for our rapid progress. 

While Greg perfects his software, 1 have designed and implemented a dynamic-incremental 
fish-eye view package for the Dandetiger display. By setting a global flag the output of this 
package may be viewed on the IRIS. (Testing of the IRIS version awaits completion of 
debugging of the new IRIS software.) The Dandetiger portions of the package have been 
tested and successfully generate the intended behavior (see the attached hardcopies). When 
a point of interest is selected in the Org-Chart display (and the function FEV is called), the 
node nearest that location becomes the focus of a dynamic fish-eye view which spreads 
iteratively in a breadth-first fashion through the net. The effect of these transformations is 
to visually emphasize the focus node and its nearby neighbors while eliding distant nodes. 
The path from the focus to the root of the net is also spacially emphasized as suggested in 
the Furnas formal description. Propagation is attenuated through a threshold mechanism 
to improve time performance. A mouse click may be used to change the focus and a new 
fish-eye formation will gradually result allowing a brief multi-foveal view. The fish-eye 
function is highly isolated to permit easy modification or replacement for experimentation. 
Currently, the function is based on the Furnas formal mathematical definition as stated in 
his report. The structure of the fish-eye software is general to permit convenient 
modifications for experimentation with new kinds of views. I have put the annotated code 
file for the Fish-Eye package in {Phylum}<Verner>LISP>Net>Fish-Eye(.dcom). 

While converting the software for the new IRIS, l discovered that the IXLATE translation 
routine did not implement conversion of the old backslash form of I RIS commands. I 
made a simple extension to accomplish this in the code, and saved the modified version for 
future use in {Phylum }<Verner>LISP>Net>lXLATE(.dcom). 

At our latest brainstorming sessions we have evolved many ideas for enhancing our 
hypermedia system including putting a control panel on the IRIS display and tying logical 
searches of the net to fish-eye views. Stu and Austin suggested devoting the bottom portion 
of the IRIS screen to something akin to a cockpit panel. One of the controls might be a 
fill-in form that initiates a logical search for particular information in the net. In the case of 
org-charts. a typical query could mean "Find and show me the Vice President of Research 
and emphasize the surrounding posts in the chart back up to the CEO." A fish-eye view 
would then be initiated with the located node(s) as a focus. These ideas are exciting and 
very promising directions of exploration. In the remaining week before I return for my 
senior year majoring in Electrical Engineering at the University of Rochester, I will attempt 
to make a start in their development. 
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I would like to thank Stuart Card, Austin Henderson, Tom Moran, and John Brown of ISL 
for again sponsoring me as a Research Intern, and NASA for granting partial funding for 
the Dandeiris-Hypermedia project. My activities at PARC with diverse projects, people, 
and disciplines have been very enjoyable and provided me with a good learning experience. 
The consulting related to PARC technology transfer, in which I have engaged for Xerox, 
has contributed to my further development of practical industry-oriented knowledge, skills, 
and abilities. The combination at PARC of high quality people, excellent facilities, and 
creative freedom renders the place an extraordinary research environment. 



Summary of Controls for Info-Net Flyer # 1 


Bruce D. Verner 
Intelligent Systems Laboratory 
Xerox Palo Alto Research Center 
17 August 1984 


This is a summary of the behavior and operation of the mouse-driven Whiteboard flyer. 

The flyer is a craft that is used to navigate and explore an information space displayed in 
3-D on the IRIS color graphics system. The flyer will gravitate toward a focus (or point of 
intertest) indicated by a yellow "flashlight" beam displayed on the whiteboard. A user may 
easily change the focus by aiming the beam with a "gunsight" located at the center of the 
screen. When the mouse-driven flyer is in use. the old keyboard code controls remain in 
effect. Although the mouse flyer provides linked control of all essential degrees of freedom, 
the keyboard may be used for supplemental adjustments. Documentation is also provided 
for additional keyed commands that were integrated into Whiteboard as an intermediate 
measure before creation of the mouse flyer. 

The mouse on the IRIS is used for the flyer, although with slight modifications to the code 
the Dandetiger mouse could be the control. All mouse inputs are polled (rather than 
queued) to provide the most immediate response to the current situation. The effect 
generated by pressing a mouse button(s) is usually proportional to the interval of time it is 
held. (Holding down longer increases velocity, continues focus selection, etc ...) 


Control Assignment on IRIS. mouse: 


Mouse Chord Behavior 


LB MB RB 
LB -- RB 
-- MB RB 

LB MB -- 


Quit the Whiteboard program 
(exits into LISP) 

Freeze ONLY Z-axis movement of flyer 
(Z-axis movement may later be resumed) 

Set Z-coordinate of focus (POI) to current 
Z-position of flyer 

(has effect of halting Z-axis movement) 

Initiates auto-homing to a viewpoint encompassing 
the entire whiteboard 


LB -- -- Select a new focus (POI) at the place on the 

whiteboard in the gunsight 
(since the flyer is attracted to the focus, changing 
the POI will affect X and Y-axis motion; just point 
the flashlight where you want to go!) 
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- MB -- 

- - RB 


more negative Z-thrust (toward the whiteboard) 
more positive Z-thrust (away from the whiteboard) 


mouse X-axis 
mouse Y-axis 


right-left head rotation 
(used to look around) 

up-down head rotation 


[Refer to the program listing for information on new additional keyboard codes.] 
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PROGRESS REPORT 


COGNITIVE AND PERCEPTUAL PRINCIPLES OF 
WINDOW-BASED COMPUTER DIALOGUES 

Our approach to understanding window-based computer dialogues has 
two main thrusts: (1) work on the theory of human-computer interaction and (2) 
exploratory programming of new techniques for graphics dialogues. In this 
reporting period we were able to construct a system for displaying trees on a 
simulated "whiteboard" in 3-D space using the project’s "Dandelris" (Xerox 
1109 DandeTiger Lisp machine linked to a Silicon Graphics Iris 1300 through an 
ethernet). The tree application chosen was to display an organization chart. 
The user was able to zoom into a particular node or to "fly" along the surface of 
the whiteboard. Several algorithms were tried for laying out the tree. It was 
found that (1) the size of the nodes could not be simply scaled to the size of the 
whiteboard because very skinny tall nodes would likely result for most 
whiteboard shapes. (2) If the nodes were scaled so that they had reasonable 
shapes and separations, then a chart of around a thousand nodes was 
indistinguishable from a straight horizontal line when the user zoomed back. We 
therefore tried displaying the nodes in fish-eye form, where the size of the node 
depends on its distance from some designated focus node and on some 
specified function for calculating the falloff of interest with distance. When this is 
done, the user must wait some time before the system can redisplay the tree after 
he has indicated a new point of focus. We therefore worked out ways to 
"pulverize” the algorithm, that is to have the fish-eye layout be changed node by 
node. The major difficulty with the resulting system is response time. An 
analysis suggested that the problem was in the buffering on both sides of the 
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ethernet. At this point we upgraded to an Iris 2400 workstation, which had the 
advantage of more graphics primitives. 

For the new Iris 2400, we built the Dandelris II, rewriting the Lisp Iris IO 
functions and adding over a hundred primitives to the Lisp side of the 
connection. We also programmed a system to compile from the C language 
version of the graphics primitives into Lisp. We reran our programs in this 
version, but determined that response time in this version was actually lowered 
by one graphic primitives Silicon Graphics had discarded from the new library. 

Experiments with C versions of our existing code showed that very good 
response was possible by breaking the system into an animator than runs in C 
and a network program that runs in Lisp. This version of the system is called 
Dandelris III. With the help of the C version, we developed a new "impulse flier" 
that allows the user to zoom in much more rapidly to portions of the tree than was 
previously possible. We are now implementing this system. We will then have 
worked out an architecture for an interface building toolkit in the advanced 
graphics area. 

Along with this progress, various theoretical issues are becoming more 
clear. Particularly interesting is the interaction between window/advanced 
graphics interfaces and intelligent systems. Application systems can be 
described by a dialect of Sheridan’s supervisory control model we call the "triple 
agent model." Different sorts of intelligent systems can be got by applying 
intelligence to different parts of the model. The triple agent model makes it 
particularly clear that there are many different ways for the user to interact 
with an system, differing principally according to who approves whose work and 
who can take the initiative when. This calls out issues of the control structure of 
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dialogues and we have begun thinking about how these relate to window 
systems under the name of visual discourse studies. 

It is clear that for very cognitive tasks such as the so-called "idea 
processing"/ "idea-browsing" systems now beginning to be seriously 
researched, advances in windowing techniques along the lines investigated in 
this project will be important. 

Describing the interaction with the user for a graphical user interface is 
quite difficult. We have therefore analyzed several interface techniques into a 
small catalogue preliminary to making formal descriptions of them and as an 
experiment in how to describe these. 

Following are a set of papers that elaborate in various ways the themes 
listed above. In this period, several students worked on the project for varying 
amounts of time: Michel Desmarais, Roy Nakashima, Lennart Lovstrand, and 
Bruce Verner. 
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HUMAN FACTORS AND THE INTELLIGENT INTERFACE 


Stuart K. Card 

Xerox Palo Alto Research Center 
3333 Coyote Hill Road, Palo Alto, California 94304 


By now, the average technical conference-goer is used to seeing the term artificial 
intelligence paired with just about anything. Whereever in the world there is a task that 
involves the least amount of human thought, someone in that profession (or his boss) has 
probably thought of applying artificial intelligence to it: Artificial Intelligence Applications 
for Business (the title of a recent book), artificial intelligence and economics (a recent 
sabbatical in our laboratory), artificial intelligence and library science, artificial intelligence 
and nursing, artificial intelligence and franchise fast foods, artificial intelligence and human 
factors. By now, none of these sounds particularly odd, we seem lightyears from the Lighthill 
report with its skepticism whether AI would ever amount to much. Partly, I suspect, this is 
because economical computing power has finally arrived at the point where there was about 
to be a large expansion in the number of computer applications anyway, and artificial 
intelligence, as used by some people, is really just another name for applications 
programming in new domains. But in the case of artificial intelligence and human factors, 
the relationship is really more profound. The most interesting case is where the two come 
together in the human-computer interface. Here, the boundaries between them break down 
so fundamentally that a new subdiscipline is emerging. The consequence, 1 think, is that 
human factors itself as a discipline will be permanently altered. 

Human factors is concerned with influencing the design of machines so that they are 
better operated by their human users. It began during World War II when substantial 
advances in the technology of war equipment had resulted in equipment complex enough that 
only if explicit attention were given to the human would the equipment be usable. for 
example, in order for a P-51 pilot to reach the lever that retracts the landing gear, he had to 
put his head down so far in the cockpit that he could not see out the window. This was not a 
good idea. The maneuver occurred very near the ground, and the resulting air crashes tended 
to support the idea that more attention placed on understanding the details of the pilot’s task 
would be helpful to airplane design. The upshot of this and like ^problems was a new 
systematic consideration of what the human operator could see and operate. Instrument dials 
began to be made more reliably legible, controls more reachable and less confusing, 
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procedures less error prone. Of course human factors has moved on since then. There are 
guidelines for displays, there are workload analyses, there are systematic anthropometry 
tables and computer-generated reach curves. All this is fine help for that original set of 
human factors concerns that were essentially sensorv-motor in nature. But much as that help 
might be, when applied to current systems it just is not good enough. The technology of 
machines has taken an even faster leap. 

THE CHANGED NATURE OF THE MACHINE 

The development of microchip technology has led directly to a set of changes in the 
machines with which humans interact. Collectively, these changes define a qualitative 
difference between these machines and those of thirty years ago. The new machines have 
greater functionality (they try to allow the user to do more things), they address tasks that 
are more cognitive, they are more complex, and they are more interactive. Bach of these 
characteristics makes more difficult the human* factors problem of designing machines 
operable by their human users. 

Increased functionality. The historical increase in machine functionality can be 
dramatically illustrated for jet plane cockpits. Big. 1 shows the number of cockpit controls per 
crew member in U.S. fighter aircraft from World War I on. As can be seen, the increase is 
roughly exponential. The same picture occurs if instead we consider the number of cockpit 
displays (Fig. 2). In fact, the trend is close to universal. One could count buttons on food 
processors, functions on digital watches, or modes on video tape recorders. 

My favorite example comes from a talk given a few years ago at the Human Factors 
Society annual meeting. A thermostat company was designing a new thermostat. During 
the arab oil embargo, people had been encouraged to turn their thermostats down at night to 
save energy. Soon there arrived on the market thermostats with two settings, one for day and 
one for night. Of course, in many households all the members work during the day, and so 
there is no need to heat the house then either. That led the company to consider another set of 
settings for the work hours during the day. But people only work five days a week, and so yet 
another set of settings is required to distinguish between the daytime settings on work days 
and on weekends. In fact, by the time the company had finished designing its prototype it had 
a microprocessor, a full QWFRTY alphameric keyboard, an LKl) display, and they were 
beginning to wonder if people would accept it as a thermostat (it could equally well have 
passed for a wall-mounted word-processor). 
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Fig. 1 . Number of cockpit controls per crew member in U.S. fighter aircraft, WWt to 1980. 






And that is only the thermostat. The same garden path of reasoning could almost equally 
well apply to the refrigerator, the coffee pot, the can-opener, the microwave oven. Of course, it 
has not escaped us at Xerox that the same evolution can be expected for office computer 
systems. 

More Cognitive Tasks. Computing systems are increasingly being applied to tasks that 
are more inherently cognitive in nature. Whereas a few years ago it was novel to apply 
computers to "word processing,” now systems are beginning to be applied to so-called "idea 
processing.” John Seeley Brown in my laboratory calls this the trend of moving from the 
processing of form to the processing of content. 

More Complex Applications. The applications we are now attempting to perform with the 
aid of computers are just plain more complex than those attempted before, especially the 
applications in daily life. The management of a flexible computer tutor, the automating of 
office work, the computer-based flight of aircraft, are ah very complicated things to do. ft is 
true that aircraft with complicated equipment on board have flown for years, but the way in 
which aircraft systems were constructed was to produce information in isolation and to make 
the pilot the integrating element in the system (National Research Council, 1982). Now we 
are seeking to build systems in which the automation will itself do part of the lower-level 
decision making and users will interact at a higher level with the system. 

More Interactive. Bitmapped displays support a higher bandwidth of interaction between 
the user and the system than the previous generation of alphanumeric displays. In the Xerox 
Star system, the memory bandwidth is about 50 MHz. The entire screen is repainted out of 
memory 39 times/sec. This would swamp most computer memories (Smith. Irby, and 
Kimball, 1982). The earlier Xerox Alto bitmapped display, in fact, required 60% of the 
machine processing. The consequence of providing high bandwidth to the user can be seen in 
the high time -density of user interaction we observe in laboratory videotapes of users. These 
higher rates of interaction are used to involve the machine more intimately in a cooperative 
process. As a user moves the mouse, the node of a tree structure displayed on the screen may 
also move, the links to other nodes turning and stretching so as to maintain the proper 
connections. This mode of interaction contrasts quite sharply with the turn-taking of older 
uphanumcric systems. 

There is no obvious-extension of results that appear in any human factors text that I know 
that would suffice for doing the human engineering of such systems. Advances in the human 
engineering of the computing systems we are now building really require additions to the 
discipline of human factors itself. It has been said that the most progress in science takes 
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place along the boundaries where one discipline meets another. Just as the combination of 
engineering and psychological concepts laid the basis on which human factors was originally 
founded, so too it now seems that the absorption of concepts from artificial intelligence, 
intelligent computer-aided instruction, linguistic discourse theory, interactive computer 
graphics, and general cognitive science are likely to help lay the basis for human-computer 
interaction. In order to make this point more concretely, it is useful to examine the notion of 
the "intelligent interface. ” 


INTELLIGENT INTERFACES 

The generic form of human-computer interaction is most evident in the case of an 
embedded computer, such as a computer used to help control an aircraft. In such a system, the 
potential for separating one part of the system that (partially autonomously) controls the 
flight of the airplane and another part of the system that mainly handles the users’ flight 
displays and controls is easy to see. An abstract representation of such a system is 
represented in Fig. 3 and will be termed the Triple Agent Model. This model is a derivative of 
Sheridan’s "supervisory control” model (Sheridan, 1984). In the diagram, the computing 
system comprises four parts: the user, a User Discourse Machine interacting with the user, a 
Task Machine interacting with the task, and the task. In a system having embedded 
computers, such as an airplane or an underwater robot, both the User Discourse Machine and 
the Task Machine may be independent computers and may, in fact, be physically separate 
from each other (e.g., Sheridan and Verplank, 1978). In a personal workstation computer, 
both could be modules within the same machine. The drive within computer science to 
separate the code for a "user interface management system” from general application code 
shows the movement toward the kind of logical separation implied in Fig. 3 (SIGGRAPH, 
1983). This model illustrates what Young (1984) has called the Third Law of Expert Systems 
(he did not propose a First or Second Law): 

Third Law of Expert Systems . Every expert system has to contain knowledge about two 
domains , ( l ) the subject area of the expert system , and (2) how to communicate with the 
user. 
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Fig. 3. Triple Agent Model of Human-Computer Interaction. Control loops possible: (1) Task is 
observed directly by human user. (2) Task is observed indirectly through sensors, computers, and 
displays. This Task Machine feedback interacts with User Discourse Machine feedback. (3) Task is 
controlled within Task Computer automatic mode. (4) Task is affected by the process of being 
sensed. (5) Task affects actuators and in turn is affected. (6) Human user directly affects task. (7) 
Human user affects task indirectly through controls, User Discourse Computer, and actuators. This 
control interacts with that from the Task Computer. (8) Human user gets feedback from User 
Discourse Machine. (9) Human user adjusts control parameters. 00) Human user adjusts display 
parameters. {Inspired by Sheridan, Fischhoff, Posner, and Pew, 1983, Fig. 4-1.) 


In fact, as illustrated in the Triple Agent Model, the "Law” is much broader than Young 
suggested and ought better be called the Third Law of Interactive Application Systems (like 
Young, we leave the First and Second Laws until some other time): 

Third Law of Interactive Application Systems . Every interactive application system 
has to contain knowledge about two domains , (l) the subject area of the application 
system , and (2) how to interact with the user . 

The term application system is taken here very broadly. It includes not only the usual 
applications of interactive computers (including expert systems applications) but also 
continuous process control, vehicle control, and embedded systems containing any of these. 

There are three agents in Fig. 3 (hence the model’s name): two computational agents (the 
Task Machine and the User Discourse Machine) and the user. Each of these agents 
potentially can have models of the other agents, including, conceivably, models of the other 
agents’ models (and their models of the other agents’ models (and their models of ...)). The 
existence of a well-developed User Discourse Machine for a system leads to the following 
observation: 

Observation / . Human-computer interaction with a system having a developed 
User Discourse Machine is less like the use of a tool by a human than it is like the 
conversation between intelligent agents. 

A partial consequence of Observation i is that the potential control and data flows in Fig. 
3 arc complicated and range all the way from a direct connection betwen the user and the task 
(the pilot looks out the window and moves the wing control surfaces manually); to subservient 
interaction in which the machine proposes a course of action then asks the user for approval; 
to complete automation where the Task Machine acts autonomously, taking action on its own 
initiative and not informing the user (the lower level control loops for the stabilization of the 
aircraft in advanced planes). Either the computer or the user can act as slave to the other or 
can act autonomously. 

Sheridan and his colleagues (Sheridan and Ferrell, 1967; Sheridan and Verplank, 1978; 
Sheridan, Fischoff, Posner, and Pew, 1983, Sheridan, 1984) were led to a partitioning of the 
processing of operator feedback control loops between a task-oriented part and a user oriented 
part by considerations of how lower-level control tasks could be automated leaving the 
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operator higher-level control tasks, hence their name "supervisory control” as opposed to 
"manual control,” the control paradigm it replaced. But we can come to similar conclusions 
based on considerations of how computer code that interacts with the user is to be 
modularized so as to obtain a neat separation from the application code, or based on how to 
implement user interface management systems (SIGGRAPH, 1983; llartson, Khrich, and 
Johnson, 1984). 

We are now ready to observe an obvious, but still fundamental, fact about the application 
of artificial intelligent techniques to application programs: 

Observation 2 . Intelligence may he applied at different sites of the Triple Agent Model , 
thereby resulting in different classes of intelligent systems. 

Artificial intelligence can be applied to the Task Machine of the system, or it can be applied to 
the User Discourse Machine both giving intelligent systems, but of very different sorts. And 
within these main approaches, different techniques could be emphasized. One system might 
play on its strength for knowledge representation, for example, whereas another might 
concentrate on rules for discourse. The difference between intelligence in the Task Machine 
and intelligence in the User Discourse Machine can be illustrated by the difference between 
MYCIN (Shortliffe, 1974; Davis, Buchanan, and Shortliffe, 1977) and WEST (Burton and 
Brown, 1982). 

MYCIN: Example of an Intelligent Task Machine 

MYCIN is an example of an Al program where the major emphasis is on the Task 
Machine. MYCIN is an expert system developed to provide advice on the diagnosis and 
treatment of infectious blood deseases. The program encodes expert knowledge as a set of 
rules, such as 

PREMISE: (SAND (SAME CNTXT GRAM GRAMNEG) 

(SAME CNTXT MORPH ROD) 

(SAME CNTXT AIR ANEROBIC)) 

ACTION: (CONCLUDE CNTXT IDENT 8ACTEROIOES TALLY .6). 

which roughly translates into English as 
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If: 1 ) The stain of the organism is gram-negative, and 

2fthe morphology of the organism is rod, and 
3) the aerobicity of the organism is anerobic, 
then: there is suggestive evidence (.6) that the identify of the organism is 

baceteroides. 

The program has no explicit model of medical knowledge. Reasoning is done by backward 
chaining through its rules, eventually arriving at a diagnosis and suggested treatment. This 
part of the program, the Task Machine in our terms, has been shown in formal evaluations to 
perform at expert level ( Yu, Buchanan, Shortliffe, Wraith, Davis, Scott, and Cohen, 197.9a; 
Yu, Fagan, Wraith, Clancey, Scott, Hannigan, Blum, Buchanan, and Cohen, 19796). 

Since lVJYCIN has to communicate with physicians somehow and since it is important for 
physicians to be able to understand how the program arrived at its conclusions, it also has a 
User Discourse Machine portion that could, in addition to asking the user to supply it with 
various laboratory and other data, respond to queries of why it had reached some conclusion. 
For example, if the system said: 

23) IS J. SMITH A COMPROMISED HOST? 

the user could query, 

**WH Y 

to which the system would reply 

[3.0] THIS WILL AID IN DETERMINING WHETHER J. SMITH IS IMMUNOSUPPRESSED. 

IF 

[3.1] J. SMITH IS NOT A COMPROMISED HOST 
THEN 

? 

IT IS DEFINITE (1 0) THAT J. SMITH IS NOT IMMUNOSUPPRESSED 
[RULE343] 
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While this attempt at a User Discourse Machine was clever and did make it possible to 
examine part of the machine's reasoning, there are severe limits to what users could learn 
about the machines reasoning that now have become clear in retrospect (Clancy, 1983, Young 
1984). For example, the machine can not justify its rules or explain why it considers some 
hypotheses before others. Hut the main point for our present concern is that MYCl.Y is an 
example of a system where the main effort at intelligence is directed at the Task Machine 
portion of the system. 

WEST: Example of an Intelligent User Discourse Machine 


By contrast, in WEST, the main effort at intelligence is directed at the User Discourse 
Machine portion of the program and relatively little at the Task Machine. WEST is designed 
to help children practice arithmetic. The program displays a route between two cities (see 
Fig. 4). On each turn, the program spins three spinners, generating three numbers for a 
player. The player can combine these numbers using parentheses and the four arithmetic 
operators. Because there are shortcuts and bonuses for landing exactly on intervening towns, 
it is often advantageous for the player to combine the digits he spins so as to make a shorter 
move than the maximum obtainable number. As part of the User Discourse Machine, from 
time to time, a "coach” pops up and gives unsolicited advice on strategy, or the player may 
himself request a hint. For example, the coach might say: 


Perhaps you have forgotten that it's OK to rearrange the numbers. You could get to 
be really good if you tried using other orderings. One good example would be the 
expression: (2*4)+ 1, which would have resulted in a TOWN! YOU would have 

ended up at 70 with the COMPUTER finishing up the turn at 54. 


Would you like to take your turn over? 
= > YES NO 


'The User Discourse Machine for WEST addresses the problems of (1) when to break into the 
game to offer advice and (2) what to say. To do this it uses an 'Issues and Examples’’ 
paradigm. Issues are characterizations of where the user is weak. Examples are possible 
moves that could be made on a particular turn. The program accumulates evidence from the 
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Fig. 4. Game display for WEST. (Burton and Brown, 1982, Fig. 2.) The gamboard above shows two 
tokens moving along a number line from starting to ending towns. If the player lands on the 
beginning of a shortcut, his token is advanced to the end of the shortcut. If the player lands on a 
town, his token is advanced to the next town. 













player's behavior over several turns for which issues characterize his weaknesses. It decides 
when to break in and gives advice to the user based on which issues it has evidence for and 
what opportunities for making dramatic points about these issues are presented by the 
possible moves the current turn provides as examples. If, on previous turns, the player has 
rather consistently missed short cuts to his disadvantage, then SHORTCUTS becomes an issue. If 
later, the player has the opportunity to beat the computer by taking a shortcut, but fails to see 
it, the machine seizes on this example as an excellent and memorable opportunity for 
coaching the player about shortcuts. 

The issues in WEST are elements of the skills required for playing the game, such as 
SHORTCUTS, parentheses, subtraction, etc. On each turn, recognizers for each issue examine the 
player's move to see whether the student could have gotten a better score if he had used the 
tactic associated with the issue. As a result of these recognizers, the system tallies statistics 
in its model of the student (Fig. 5). Evaluator routines work on the student model to decide on 
which issues the system can determine that the student is weak (the evidence on a particular 
issue may be inconclusive at a given time). Given the issues on which the student can be 
shown to be weak and given the possible moves available on a given turn as examples, the 
system proceeds according to a number of principles (Burton and Brown, 1 982): 


Principle /. 
Principle 2. 


Principle 3. 

Principle 4. 

Principle 5. 
Principle 6. 

Principle 7. 

Principle 8. 

Principle 9. 


Before giving advice, be sure the Issue used is one in which the student is weak. 
When illustrating an Issue, only use an Example (an alternative move) in 
which the result or outcome of that move is dramatically superior to the move 
made by the student. 

After giving the student advice, permit him to incorporate the Issue 
immediately by allowing him to repeat his turn. 

If a student is about to lose , interrupt and tutor him only with moves that will 
keep him from losing. 

Do not tutor on two consecutive moves , no matter what. 

Do not tutor before the student has had a chance to discover the game for 
himself. 

Do not provide only criticism when the Tutor breaks in! If the student makes 
an exceptional move, identify why it is good and congratulate him. 

After giving advice to the student , offer him a chance to retake his turn, but do 
not force him to. 

Always have the Computer Expert play an optimal game. 
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MOVES 

(A + 8)*C 

(A*B) + C 

(A v B)-C 

(A*8)/C 

A/(B*C) 

A*(B-C) 

(A*3)-C 

8est 

1 

3 

- 

- 

- 

- 

- 

Good 


1 

- 

- 

- 

- 

- 

Fair 

- 

1 

- 

- 

- 

- 

1 

Poor 

- 

1 

- 

1 

- 

- 

- 

MISS 

2 

1 

1 

1 

1 

1 

- 

SPECIALS 

Took 

Missed 






Town 

3 

2 






Bump 

- 

1 






Shortcut 

- 

- 






DIRECTION 

Good 

Poor 

Was Best 

> 




Forward 

5 

a 

9 





Backward 

- 

- 

- 





SPIN-ORDER Good 

Poor 






Original 

2 

3 






Reverse 

- 

1 






Decreasing 

1 

- 






Increasing 

- 

- 






Other 

2 

- 






PARENTHESES 








Need 

Use 

NO 

None & 





& Use 

Anyway 

Use 

& Miss 





1 

- 

8 

3 




STRATEGY 

Max 

Max 

Max 

Specials 

Other 




Delta 

Dist. 

Number 






3 

4 

2 

3 

4 




Fig. 5. WEST Model of Student. 



Principle 10. If the student asks for help, provide several levels of hints. 

Principle 1 1 If the student is losing consistently , adjust the level of play. 

Principle 12. Tf the student makes a potentially careless error, be forgiving. But provide 
ex plic it co m m enta ry i n ca se it u>a s not ju st ca r el e s s . 

WEST thus represents the invention of a considerable amount of mechanism in the service of 
the User Discourse Machine: conversational turn-taking (when it should break in), 

explanation, diagnosis from user behavior of the state of user knowledge, and even user 
motivation and emotions. By contrast the Task Machine (computing the best move on each 
turn) is relatively straight forward. 

The application of intelligence to the User Discourse Machine is my provisional choice for 
what we mean by an intelligent interface. Such systems can come in a variety of forms: 
intelligent front-ends to powerful, but hard to use, applications programs; advisor programs; 
coaching programs; programs that allow natural language interaction. Whatever the exact 
form, the clear intent of all such applications of Al to the User Discourse Machine is to make 
systems easier for people to use, largely by attempting to deal with the cognitive or linguistic 
content of the communication task with AI tools. But making machines easier to use lies in 
the center of human factors as a discipline, as well. For this reason, progress on the AI end 
will not leave human factors unchanged. Either human factors will absorb into itself the 
concepts from AI that make interactive machines more possible, or it will cease to be the 
major discipline concerned with this area, or, an interesting possibility, there will emerge a 
new subdiscipline of human-computer interaction at the intersection of the several disciplines 
with potent contributions to make in this area. This latter is an attractive possibility because 
there are other disciplines whose recent results are also important: Interactive computer 
graphics is concerned with algorithms underlying graphics interfaces, computer science itself 
is concerned with developing "user management dialogue systems,” linguistic discourse 
analysis is concerned with conceptual analyses of conversational interaction. 

PROBLEMS OF THE INTELLIGENT INTERFACE 

It must be said that the design of intelligent interfaces is difficult and not well 
understood. We have said that instead of thinking of a human operating a machine, it is now 
appropriate to think of communication between intelligent agents. This means the machine 
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no longer does necessarily what it is told to do, but what it thinks the user's intention is, or 
even what it thinks ought to he done if it knows the user’s intention is otherwise. I have 
stressed the impact of developments in artificial intelligence and intelligent tutoring systems 
may have on human factors. Young (1984) has made the other side of the case, arguing that 
interactive Al programs need to pay more attention to the human factors of their interfaces. 
These two arguments are not incompatible. As we have seen, it is possible to build Al systems 
that concentrate on the Task Machine giving little attention to the human interaction. I will 
now argue that there are still a considerable number of problems to overcome in the 
technology of constructing intelligent interfaces. These problems are important limiters of 
what presently can be done. Work on them forms part of the research agenda for human- 
computer interaction. Three examples are: the role problem, the automation problem, and 
the communication problem. 

The Role Problem. Young ( 1984) has pointed out that an expert system can be designed to 
function in a number of roles vis-a-vis the user. For example, he points out that MYCIN is 
strongly committed to being the partner who actually makes the decisions. This choice of role 
may influence the acceptance of the program bv users quite independently of the programs 
actual performance expertise. WEST explicitly adopts the role of coach; in fact, the research 
question for WFST is really, how can the role of coach be modeled and implemented in a 
machine? The question of role is related to the question of persona. How should the system 
present itself 7 Should it have the persona of a person or a tool or an environment? MYCIN 
and WEST both try to establish personae as artificial people. MYCIN uses natural language 
dialogue and a heavy dose of medical mannerisms to establish the persona of a consulting 
physician. WEST uses a small cartoon picture and breezy English dialogue to establish the 
persona of an athletic coach. 

The Automation Problem. There is a long history of problems that have arisen under the 
name of automation (National Research Council, 1951; National Research Council, 1982), 
especially in connection with the automation of aircraft control systems, since it was in this 
area that systems of limited intelligence first occurred. The problem is often stated in the 
form of, which tasks should the human do, and which the machine? As automation work 
spread to underwater robots and teleoperation, similar sets of problems began to he studied 
under the term 'supervisory control.” In the case of intelligent tutoring systems, Al expert 
systems, and even the general human-computer interface, we again see a familiar set of 
problems arising, this time in the context of discrete symbolic systems. 'The possibilities 
inherent in the combinations of control loops in Fig. .1 result in a fair-sized design space of 
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automation alternatives. 'This space has much more structure than the simple division into 
machine-initiative, user-initiative, and mixed-initiative categories (Carhonell, 1970). One 
set of possibilities identified by Sheridan and Verplank ( 1978) is given in Fig. 6. The list runs 
the gamut from direct control by the human (corresponding to paths l and 6 in Fig. 3), to 
complete control bv the machine (path 3 in Fig. 3), to some of the many combinations of other 
paths. 

The list serves to indicate how complex the possibilities are. In fact, even more 
possibilities are possible that are not on the list: The words human and computer can be 
reversed. WEST presents still another possibility, the human selects the option, but the 
computer gives advice if a sample of the user’s decisions suggest it is needed. ( M VC IN is^ an 
example of option 4 above, the computer selects the action and the human may or may not do 
it.) If we distinguish the computer (or human) explaining why it did something from mere 
telling that it did it. we have vet another set of possibilities. This puts us beyond the older 
form of the automation question, which tasks should the human do and which the machine? 
and into the much more complex consideration of how should they cooperate? The question is 
not dissimilar from the question of how can humans be organized to cooperate on a task? 

The Communications Problem. It is not surprising that many issues of the intelligent 
interface end up under the category of commications problems. First, there is the issue of 
trying to find a framework of interacting with the user that is open, but natural and 
predictable. This desire leads naturally to attempts to apply and generalize linguistic 
discourse theory so that it applies to computer interfaces. For example, Grice ( 1975) has put 
forward the notion that human conversation takes place in a number of "conversational 
moves,” each taking the conversation to a new state in the discourse. Grice formulates a 
number of maxims and defines inappropriate conversational moves as moves that violate 
these maxims. Reichman-Adar (1984) has taken this approach further and attempted to 
describe, using an augmented transition-state network, a way in which conversational moves 
may be mechanized. Such discourse notions as conversational focus; control of turn-taking; 
opening, narrowing, and closing a conversational context; time scale: and deictic (pointing) 
reference need to be examined in the context of computer-based dialogue. 

Second, there needs to he more work on modeling the dynamic, communication induced, 
changes in the user. In order to carry on an intelligent conversation with the user the system 
must he able to diagnose in an approximate way the user’s state of knowledge about the topic 
of conversation (WEST spends considerable effort attempting to do this) and the system must 
contain the implementation of a theory of explanation. 
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1 , Human does the whole job 
up to the point of turning it 
over to the computer to 
implement. 


2. -Computer helps by 

determining the options. 


3. Computer helps determine 
options and suggests one, 
which human need not 
follow. 


4. Computer selects action and 
human may or may not do 

it 


5. Computer selects action and 
implements it if human 
approves. 


COMPUTER 


(GETS options from outside) 
SELECTS action 
STARTS action 


(REQUESTS notions) 


SELECTS action 
STARTS action 


(REQUESTS options) 


(REQUESTS SELECT action) 


SELECTS action 
STARTS action 


(REQUESTS ootions) 


(REQUESTS SELECT action) 


APPROVES SELECT action 
STARTS action 
if HUMAN APPROVES 


(REQUESTS options) 


(REQUESTS SELECT action) 


APPROVES START action 




GETS options 


SELECTS action 



GETS options 


SELECTS action 



GETS ootions 


SELECTS action 


STARTS action 
if HUMAN APPROVES 


6. Computer selects action, 
informs human in plenty of 
time to stop it. 


(REQUESTS options) 


(REQUESTS SELECT action) 


APPROVES START action 


GETS options 


SELECTS action 


STARTS action 
.f HUMAN APPROVES 
or if t > r and HUMAN 
HAS NOT DISAPPROVED 


7. Computer does whole job 
and necessarily tells human 
what it did. 


(REQUESTS SELECT action) 


GETS options 
SELECTIONS action 
STARTS action 
TELLS action 


8. Computer does whole job 
and tells human what it did 
only if human asks. 


(REQUESTS SELECT action) 


(REQUESTS TELL action) 


GETS options 
SELECTIONS action 
STARTS action 
TELLS action 
• f HUMAN REQUESTS 


9. Computer does whole job 
and tells human what it did 
if it decides it should tell. 


(REQUESTS SELECT action) 


GETS options 
SELECTIONS action 
STARTS action 
TELLS action 

• f COMPUTER APPROVES 


10. Computer does whole job if 
it decides it should and tells 
human if it decides he 
should be told. 


(REQUESTS SELECT action) 


GETS ootions 
SELECTIONS action ' 
STARTS action 
if COMPUTER APPROVES 
TELLS action 

.f COMPUTER APPROVES 


Fig. 6. Alternative levels of automation possible in human-computer dialogue. (Redrawn from 
Sheridan and Verplank, 1978) 











































Third, the way in which communication with the user interacts with the constant visual 
presentations available to him needs to be better understood. The graphics display allows the 
system to make a conceptual model compelling to the user and to give memory support for the 
components of that conceptual model. Furthermore, the user can specify bv pointing or 
gesture things to the system that would otherwise be difficult to express. 

HUMAN FACTORS AND ARTIFICIAL INTELLIGENCE 

Communications with future systems will surely be more subtle than with today’s 
systems. Mouse-pointings, gestures, command buttons, typed commands, voice, voice 
intonations, and eye gaze are all input modes within the current state-of-the-art. 
Explanation, diagnosis, and negotiated information retrieval are some of the modes of 
interaction desired. The human engineering of these systems will only be possible by 
collecting the results from sciences outside the traditional sphere of human factors. 

Both human factors and artificial intelligence have disciplinary interests in human- 
computer interaction. In artificial intelligence this interest has been expressed in work on 
the intelligent interface. Theoretical work in the related disciplines of intelligent computer* 
aided instruction, computer graphics, discourse linguistics, cognitive psychology, and 
computer science also look relevant. 

Just as the combination of engineering and psychological concepts laid the basis on which 
human factors was originally founded, it may now he appropriate to attempt to lay a new base 
for human-computer interaction based on these cognitive and computer sciences above. This 
could be done within the human factors discipline, but it could also occur largely outside of it, 
in a new subdiscipline of human-computer interaction. Whichever is the final choice, Al and 
the intelligent interface are sure to have permanent affects on human factors as a field of 
study. 
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INTRODUCTION 


The technology of bit-mapped graphics and mouse pointing devices has made possible the 
development of a mode of interaction which is clearly different from interfaces based on 
teletypewriter technology. This new interaction style relies heavily on the use of graphic symbols 
such as icons, metaphors such as windows, and command selection by pointing at menus. 

Recently, the public’s acceptance of the Apple Macintosh personal computer has made it clear 
that this new style of user-interface design is becoming more universally accepted (Markoff, 1984). 
Systems based on this style have been in use at PARC since 1974. Many years of independent 
development has resulted in a surprising amount of diversity among them. New systems are 
continually being developed through a combination of innovation and the borrowing and 
adapting of ideas from existing systems. 

Designers of new user interfaces face several problems, mostly related to communication. One of 
these problems is maintaining an awareness of what has already been designed. Ignorance of 
existing design solutions can result in wasted effort in "reinventing the wheel". Deviations from 
existing conventions can create compatibility problems with other systems. On the other hand, 
exposure to the work of other designers can act as a stimulus to creativity. 

The second problem is that designers need to be able to communicate the stylistic aspects of a 
proposed user interface design without having to implement it on a live system. There are several 
reasons why this is useful. During the early stages of a project, it is desirable to generate and 
evaluate a large number of designs economically. The generation of a large quantity of 
preliminary design ideas increases the probability that a good idea will be found. The 
performance of at least preliminary evaluation of ideas or concepts before implementing them can 
reduce the amount of effort wasted pursuing inappropriate design directions. Once an idea or 
concept has passed preliminary evaluation, they must be communicated to programmers for 
implementation. All the stylistic, interactive aspects of the design must be described completely, 
yet concisely. After a system has been implemented, designers may want to document what has 
been done. Documentation specialists may wish to describe the operational aspects of the 
implemented system to novice users. Clearly, a method of efficiently communicating the stylistic 
aspects of user interface designs would be valuable to many computer professionals. 



The third problem is that there does not yet exist a comprehensive, generic vocabulary of user 
interface techniques. Because of this, it is difficult for system designers from different companies 
or backgrounds to converse or discuss user interfaces without having to refer to specific products. 
It is hoped that, by cataloguing the many techniques which have been employed to date and 
determining a taxonomy for these techniques, we can move one step closer to the development of 
a generic vocabulary. Other possible benefits of such a taxonomy include the development of a 
methodology for selecting the best interaction technique for a given application and a directed 
exploration of the design spaces which are found to be underrepresented in the taxonomy. 

This document contains a partial catalogue of interactive graphic techniques presently used to 
implement user interfaces in a variety of commercial and research computer systems, including 
Interlisp-D, Star, Smalltalk, Cedar and Apple Macintosh. It combines text and graphics in an 
innovative way to provide an effective means of communicating the interactive, stylistic aspects of 
window system designs, both existing and proposed. It has also proven to be a useful tool for 
documenting user/computer dialogues for further study by interaction languate analysis. 

The catalogue was compiled during the summer of 1984 while the author was employed as a 
summer research intern at Xerox Palo Alto Research Center. It is by no means complete and is 
intended mainly as a demonstration of an effective communication and documentation 
methodology. The author wishes to thank Stuart Card and Bill Verplank for their guidance and 
the initial idea of writing this document 



DESCRIPTION OF THE CATALOGUE 


Organization 

The catalogue was organized around relatively high level goals such as "copying a document" or 
"moving an icon" because it was felt that the quality of the user interface is determined more by 
the way in which elemental acts combine to form a command sequence than by the elemental acts 
themselves. The approach was taken that the best user interfaces are designed so that the sequence 
of events or actions required to specify a particular command form a gestalt which enables the user 
to fuse them into a single act 

Presentation 

The catalogue has both graphic and written content. Although it is obvious that graphic content is 
necessary to communicate the graphic quality of an interface, supplementary written explanations 
are also required to keep the overall format as concise and economical as possible. A picture is not 
always worth a thousand words and it is sometimes easier to communicate a message with words 
than with pictures. 

The catalogue shows both sides of the human/computer dialogue. The idea of conversational 
interaction with computers has been around for some time (Nickerson, 1981). In order to 
accurately depict and evaluate the quality of this collaboration, it must be possible to show with 
equal clarity and detail the actions and responses of both the user and the computer. 

Grain of Analysis 

The temporal grain of analysis was selected to match the experiences of the user. Each command 
sequence was broken down into a number of sequential frames, similar to those used in comic 
books or cinematographic story boards. One frame was provided for each distinct visual event 
experienced by the user. E.g., if two visual events occur so close together in time that they are 
perceived to occur simultaneously, then both of these events are depicted in a single frame. 



The spatial grain of analysis was selected to match the visual characteristics of the user. It was 
assumed that the user’s attention would be focussed on a relatively small portion of the screen, 
therefore in most cases only a fragment of the screen is depicted in each frame. Judgements 
regarding frame content were based purely on the subjective experiences of the author. For 
example, when describing the operation of the Star system, the function key pad is shown each 
time a function key must be pressed because the author never acquired enough proficiency with 
the system to memorize the position of the keys and manipulate them by touch, whereas the 
mouse is not shown each time a mouse button is pressed because the author never felt the need to 
look at the mouse. 

The narrative content of the each of the frames is numbered to correspond to the exchange of 
conversational control between user and computer. Each numbered narrative segment describes 
the events which take place during one complete exchange of conversational control. Each 
narrative segment begins with a decision or task to be performed by the user and ends with the 
response provided by the system. The narrative segment begins at the point in time just after 
conversational control has been transferred from computer to user, includes the events proceeding 
and following the transfer of control back to the computer and ends at the moment the computer 
relinquishes control again. 

As might be expected, there are a few exceptions to the rules described above. In order to 
accurately depict the subtleties of mouse techniques, it was necessary to choose a finer grain of 
temporal analysis during mouse buttoning activities. Although most users experience the clicking 
of the mouse button as a single event, in some cases it was necessary to illustrate this action with 
two frames in order to show events occurring while the button was being held down. Occasionally 
it was necessary to provide separate frames for visual events which occurred simultaneously, but 
were spatially separated. This can be justified by considering that only one of them is in the user’s 
field of view and is obviously perceived first. The other is seen only when the user shifts his gaze 
and is therefore perceived as temporally separated from the first visual event, even though the two 
events occurred simultaneously. When this type of situation is presented to the user, it is depicted 
using two frames instead of just one. During cursor positioning tasks, the user is obtaining 
essentially continuous feedback from the computer throughout the performance of the task. 
Strictly speaking, there are hundreds of interactions occurring between user and computer. 
However, in order to keep task descriptions as concise as possible, and because it can be argued 
that the task is experienced as a single event, throughout this catalogue, cursor positioning tasks 
are illustrated with a single frame. 



NOTES 


1. Because of problems of compatibility of screen graphics data between the many different 
computer systems, it was decided that traditional graphic design techniques would be the 
most efficient way to generate this document (as opposed to more advanced electronic 
publishing methods). These include the creation of original artwork by hand, the use of 
variable reduction photocopiers to produce repetitive images of the required size, 
modification of photocopied images by hand to save time using opaque white paint and 
black felt tip markers, cutting and pasting photocopied images. 

On the other hand, it was found that the use of hardcopied bitmap representations can 
greatly speed up the process of screen depiction, especially when the images are repetitive or 
finely detailed such as pop-up menus, reverse video images, or window contents. If 
possible, hardcopy bitmaps should be printed at larger than full scale, since most of the 
frames are depict screen images at greater than full scale and because reducing photocopiers 
are easier to get access to than enlarging photocopiers. 

2. Commercially available dry transfer media such as "Zip-a-tone" should be used to render 
"gray" textures used in backgrounds or scroll bars. Hardcopy of the actual bitmap used for 
the texture can also be used. Gray felt tip markers are not recommended for this purpose 
since they do not reproduce well. 

3. When composing frames, the same care and principles should be followed as in composing 
a painting or photograph. Whenever possible, contextual cues should be included in the 
frame to provide the observer with some sense of the scale and screen location of the 
depicted view. The cursor and the edge of the screen are examples of good contextual cues. 

4. Although the technique used in this catalogue is an effective means of communicating the 
interactive and graphic aspects of direct manipulation window systems, it has the 
disadvantages of being time consuming and requiring a certain amount of artistic skill. 
Clearly, a more efficient, "short-hand" version of this methodology would be useful to 
computer professionals as a quick and informal way of recording observations of 
user/computer interactions. 

One example of how this might be accomplished is shown in Appendix A. In this example. 



provided by Stuart Card, a keystroke model is used to record interactions between user and 
computer. Once the interactions have been reduced to this form, they can be further 
studied using interaction language analysis. These analyses allow researchers to make more 
meaningful comparisons between window systems and may lead to new insights into the 
nature of interaction languages. 

These notations can be generated either by direct observation of a user or from inspection of 
the diagrammatic representations of user/computer interactions contained in this catalogue. 

In the example, P refers to a cursor positioning task, K refers to a keystroke, M3 refers to 
the third mouse button. 
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OF POOR QUALITY 


51 tfelea <£. mouse- buMt™ . 

owfiwe ctfSAppeAKi * 

cwcrwvuA Of&y 





'MSg£Tk3M V0\urr agupcttoN 


WV I 

t 


the Hrd 


■r 

f 


ffcsrftevu UAfior on cMaraokv' 
imi v&diCvfeAv pre££&[iVvi & 
de&freA i\^eWi>vi peiVvr. 


2. Vress 

Cfaraofer r&se*s2.s. 

CM-ek abpeA'TZ 

eJz&feX- ' (Marau^r J 


4 



TASK'. 

IWSE^Tl^K) R^IOt ELECTION 

SYSTEM •* 

MACINTOSH 


v 1 . - 


Posrtfcm cursor cm cte&mt 
foseWlfcw pxVvh ftess rywvtse. 

bicthm. 


Flashfrvj veAicaJ Veor ayjxari 
Ik Mrired Cn£erii > vt 


<feNP> 



7A5K ' SBLBCTiO^ 

Si&T&A STAR £ WTtKPP l) 


the word is 



/. $?»Hpvv cwrscrc uv\4ejr 

-ftaf tetter fVi ward. 



2. 'flfis&s I * 23 **“ (otfH<!>V\ 

ffiew^sc- vu^c >3 


the Oord is 


3. posrhVvt cursor iwv^ter 
lash letter fa wtnrd. 



4 n^M" Isvtfen 

C Averse, vd^ -J 

<feNP> 



WOKV SELECT I CM 
ZYSrerf < AK 1 ( METHOD j£) 



I. fos'dlenn. CMhS<n~ uvuie** a* iy 

ic^ie< i* w^rd. 


the wHrd is 

t 


-2-. ft«*S lePf Jfarwr 2 le>urtt«M . 
[ Reverse, vi 


the 


T is 


Ktfr^. iS 


\sPc 0$ tvun^&e. ip«t tW. 


I [fceve*tfe. Vnteo’] 





1*5K > w C#P 

macintosh 



ord is 


I. PosffjcTvv cursor Jeff of 
£0ider first letter iw 
wonA. 


thej^o 


rd is 


2-. PVess arud KdU rvuttse. 
t?uth?v\. ^IBF VeHt*c<al kwr" 
*fc> left of ^vs4" 

Wter. 



the BP^jrd i 



faletee wifiusc ipu$0* 

evvtfee w*» w»^ f* rzveneJ. 

<&ip> 




TA^C 

£U2SIM£ A WlKlPOW 

SYSTEM: 







Close 


I- f ’oeetiov^, CMYZcrr 0+. ^Ue. "ciaes * 

J oac iVt ~h\e. -fbjp left* Cavne* o{- 
iW wTv^^vt*. 


2 . Ftess le£f YVUPH&& Lid* v^. Afc 

fctftUt Ot pwgfia<, "CUSSE* Jp«9< 
revciges vriHe^. 




2- (cadruiecl) fie b*riW is 
YeieOiad, cwrecr cWw vye& 
Mr^66, Tffie k**r Q\ri\rk&. 


2 . (amkru+eA . ) MwW- di sappmrs . 
Cvcrecc veu&ns \z> nomw*4 shape.. 
<BMP> 



ICON OPERATIONS 



TASK- 

CDP/tNfg A VOCU M^sTT 

BASE / 0E 2- 

SYSTEM: 

gw? 



PosHiOh Cursor on desired 

docwne uvt icon. 


5. fVssg, left" mouse, button. 
£ Icon reverses vfdeo-^ 


PELtrlE- 


Ffess “COPY" ktdt&n. 

£MeSSfc^2. appears in jorany^i" 
to/wteu; ; Ptsase i>ritode a 
de&hnahon uJto either Wffem ./ 


ConHnued. ( loon reverfc. +» 
r^rvnai video. Cursor ckxK 
tVtfo docume^ mini -/'con 
dtofe-- 




TA&K i 

C 0 PY 1 N& a p^^mekjt 

f&3E 2 OF 2- 

SYSTEM: 

- 



Postkiovx cursor m <tec\re£ 
\ocaH&n. 


Phss 6 either rrouse. button. 
(VlhAe. buti&r is h eid, dccu- 
rr&ut ia>v\ appear* laehind 
cursor.') 


VocJjm 


*5. Covrkmed. (_Mten button it 
y^ieased , cursor eh owcps to 
hour Qiaos <sJlnczpe..' s ) 


C. <^urffKvu€tfl . CAftor ’foua 
secorck, icon copy reverses 
vfcteo. te> 

ncwmJ skiope.') <ENP> 





TASK*. 

SYSTEM' 


cow iw£ a poamaoT 


fites i cp a 







TASK s CQF/INS A D OCUMENT 
SYSTEM : MACINTOSH 


m<3e 3 . of <a 


Edit Uieup Special 



Paint Work 


Dr 


4 

\j 

Mood 


Get Info ^381 
Put Back 


Close 
Close Rll 
Print 


pr 


E )e< S 


E SQ 


329K in di 


Oram 


26K in 


JO. 


£>. f befHo\ cur&r on "P upliode. 
C/vnrnand. ( ''Pu^Ucyvk 11 flzvexsek 
v^eo.^ 1 




[ « File Edit Oiem 

Special | 

Paint Work 

4 items m 

329K in di« 

V 



System Folder 


/3 

=n==== 

... . = = 

__i_ Drain 


4 items 


26K in 

MocF 

JD_ 

1A_ 

on 


(?. %£te&6e~w0 Mgfe btdfevi. ^Atevtu 
Curse* CMcUAALa 

iVte> * 


v / 






0 I*q uj joy ^ ■ ■ ■■ 

32K in folder 

64K a 



Close Box 


B 

Keyboard 





ORIGINAL PAGE IS 
OF POOR QUALITY 


6 continued . Coffer a -few seconds^ 
icon reverts, tt> normal video . Kreur 
fdon appears m reverse, video , 
-to lewer rvgkt* oP original. Cursor y 
r&/eurte -b norvnol shc\pe ") < 000^/ 

<&jvy 


J 


33 . 




& A- VOCUM&tf T 
STAtf 


carsorovi de&lred 
docum&wt icon. 


ORIGINAL 

OF POOR QUAL5TY 



Press leff mouse. IpuifcsM. 
reverses video.') 



ts^rsnf 





Ffess loutiavi. 



3. 0>nj^naed. (_ t^on raveHs b< 
nonnaJ vifcleo. Curse* cfoewCfS 
•b hourglass Sli^oe..') 





P^AMINJS A 



4- C?rcferties window appears in 
batom right- Corner at Screen . 
Cursor reverts *t& rvorwei slttjpa. 


4- <30^hViuec( . ( Pooiwewf- rra**«e. 

appears Cm reverse. \ndee> awt 
IS -followeM ley blfnbiW^ Caret.) 

^ J 


■S' 




su 


raagm=tffl b| 

0. frTg^ l lS2H5 


Name 

S^noH 


COVER SV 


Version of: 06 / 27 / 84 - 17:56 

Size as oA Lash f^ginarte. 


Ql 


5. Press "P^LETE" button. 


5". a&mnued ( Pocuvnewh name. 
Is deleted. 





JR^NmiNkS ^ voom gK/r 




DOCUMENT P 


1523513 


Name. New Name^ 

Show [COVER SHEET] 

Version of: 06/27/04 17:56 

Size as of- Last* f&glnarte. 



ig gy 


Ma^rr P 


1^55 


New Nam&A 


COVER S 


Version of: 06/27/04 '7:56 By 

Size as of- Last' ffcginarte. 


Type in mew name, on keyboard 
( New name appears in 
fVDpeWtes. Wfnriovy.) 


7. Posrtton cursor on "PONE" box. 


RXWMENT 


© Done t£S53L' CE£ 


Name [New Name^ 


Show 


[COVER SV 


Version of: 06/27/04 '7:56 

Size as o£ Last- feginste. *. 1 


a enu g^i igj 5 Ks 


Name I New Name>y ' 
Show [COVER SHEET? 
Version of: 06/27/04 »7:56 

Size as ot Lash ffcginarte. *. 1 


Press mouse. 

(As bidfow fe pressed, k PpiC 
locpc reverses videa.) 


0 . Burned . ( As bidfon. fs 

released^ cursor <fo 

hourglass shope.) 
















£ENJAMINJ£ A VOCUMBhST 
M^-IKiTOSH 


ID 


4 items 


Ora 


26K in 


4 items 


Oraii 


26K in 


Size Box . Cursor shapes Close Box 


Size Box 


Close Box 




Cursor oa desired 
cbcume+ot rc&*. 


2 . Press tncv&e. button. (l con 
reverses vx'deo,^ 


Size Box Pointer Shapes] Close Box 


3. Type new HA*ne- o*. keybv&rd. 
(did na»ne is deleted at •Arsh 
Y&fsbroVe avd new- wwe. 
i arrears \AMder iom. } 
















*fe:< : 

CRS&Tllsf6 A NBN P^a^MeMT 

I <2F= £ 

5Y6T^M : 

star 



I . (l f Blank Paawi^wf i'cow % 
on (jfeslctep, '¥n&*\ skip *fe 
F^ovne 15.) Posrtfov\ airsor- 
on Pinec^aiy !fcoy>. 


2. FVess lePf mouse, button. (lean 
reverses video.) 


S BE^iagf 

BHBB 


2 

OSBU 


Press "OPEN" button. 


3. corttfrvued. ^Cursor d'tonges -fo 
W glass shape., loon ^appears 
leaving "Me",) J 





TA6K i 

/4 M^vv/ vgouYA^rr 

2 OF (? 

5Y6TEM- 

^tAK 



3 . cortfinued. (Wncbw appears 
on left* side. ©P screen. Cursor 
reverhs -fc normal shape. ") 


m fbs/fton cursor on line labelled 
"Basic &xu Folders a * id 
Record Rt*s>. “ 


e wgn 


NAME 


i p=j \ Basic Documents, Folders, ar 


SB]' **<"5 


jgjjjl MaMnq 

gj PH " tlV '9 




5 . FVesg lepf mouse bufron, Cbine 
reverses video.') 


6 . Pfef 6 “OPEN* Ipm-uJ 








PA&a. z op b 


task •. ci&rx\ti& a nw ux-uwetsr 

5Y6T^M •' _ 
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OF (? 


task i A 

SYSTEM : 




*E=T 


ISSS 


9. Press "COPY" key 



j 


e ass' 


Close All 


NAME 

Q Plank Graphics Transfers Cfec 


Q Blank Polder j 
O Blank Mall Note. 


9. continued. C Cursor changes. 

shojoe. M ecoay appears ih 
provujob unYvdowj "Please ihottaxfe. 
A deoftnoWon wtfk edfcer button. 1 


% 


OSBU 


v 


Ctocum 


0S8U 


1°. PosfHon cursor in des/red 
loca+fon. 


II. Press eittaer mouse button. 
(While button IS held, rcon 
(XfBpeors behind cursor.) 


ft. 








TA$K : 

CfeftT'NG A NEW DOCUMENT 

PASe S OF (p 

SYSTEM : 

57/3 P. - 



OSBU 


Blank 
| Docum 
1 ent 


OSBU 


II . continued. C When button is 
reXeo&ed^ cursor cIaaw^js -fa 
hour ylA.sc shape..") 


12 . After 2, seconds, icon reverses 
\ndee. Cursor reverts 
r\prrv&l shape.. 


e 0ES3 


Close All 


& 


Q Plank Graphics Transfers D: 
Q Blank. Document 
Blank Mder 
|g| Blank Mall Note 


12. continued. (Lire reverts tt> 
r&rmcU video.) 


Q Plank Graphics Transfers L>x 

\ 

Q Blank Document 
p=j Blank Polder 
I3» Blank Mail Note 


13. Vc&mon cursor c*\ lt Clo&. 0 








CMf(X\H& A New V OC M M£N T 
•STAR 


fASe ^ Op£ 


6 053 Close All 


NAME 

Q Plank Graphics Transfers 
n Blank Pocwmerft 
I Q Blank Polder 
I3> Blank Mali Note 


(4. B^sg ejmer mouse w&ctidvi. 

butfbn rs pressed, ''Close- 
/Ail* I oex. r&t&rees video. ^ 


|4. eorrkrioed. (As button fs released, 
window; disappears, PiVecfery ica 
reappeois fV> reverse video, 
fbcumewf Icon n^erts r&rvwJ 




\S. Maiie a c&f>f cfi 44 te. Slavik. 

rename, if. 

^Sae. (^opyiVua a Pocu**iewf 
tfen&Miyiq a. 





TASK i 

OREfert "INKS A- KJ&M p VCUWBUT 

P&SE2 l <9F 3 

SYSTEM: 

macinjt^sh 



Write/Paint 


377K in disk 




I I ^ 

Ider System Folder Moc Write 

i i 

ver Mac Advice MacPaint 


22K av 


* 


■ 

IDrite/Paint 



■III Tl 1 

377K in disk 

22K avai 


ider System Folder ifigtarma 

i <$* 

/er Mac Advice MacPaint 


Posrfvvi tfvi desired 

application icon. 


2 . Ftess 'fWe- tviAA^e l^u^cvt 'tu/Vaa , 
(pn#e$v«s<- press , \tcw 


* Fiie Edit Search Format Font Styie 


Untitled 
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2. Csvdfruieei. ( 0* -tti& second pregs, icon G/cw% iyrfo afpkcaMov i 
voi \ndow-') ^ 











*T MX- 

SYSTEM: 






pAGB 2 OF 3 



ORIGINAL PAGE- IS 
OF POOR QUALITY 









3 £F 3 


TASK i n?E£T\W( => A NEW VOCU MBtJT 
^ysreM: MACINTTQSM 


~>v 



7 T^pe. mvne. cvl 

tcgy'las^rd. ("SAVE CcntftMcwd 

-tums keypress. 

^ Ppowi^u^ appecurS 

prunpt ia>iVu*put. 


Saue document as: 


New Document| 




Cancel 


7. ftess manse. htCttcM. ( '’SAVE 
reversed \jtde&.') 


Saue document as: 


New Document] 


[ Saue ^l [ Cancel 


t- ?OZiticM. UUfSS'T CAA. 

ccrmwwyjx. 

" SAVE ” 

\ 

J 


[.rjuMMi /Paint 

\ 

377K ia disk 

22K ava 


] 



Ider System Folder Mac Write 


sSSr 

-nr-r: 



War 

A 


/er MacAdvice MacPaint New Ptxument 


V- CtwHraiect (P^y>[[C&Avm cwd 
fymyft' i/uvVicW^. iMsa^penr > Me^ 
icaw. appears tVi scw^e wMduiu as 


47 . 







TA6K * 

W9VIN1 <j AN IC^H 


PA£^ l OF 2- 

SYSTEM •• 

STAK 




Letters 


V 


Sdes n 

rJ 

Letters 


Position cursor cm de &\ red icon. 


2. Press left* buTtfon, 

Icon reverses video. 




letters 


3. Press “ MOVE" button. Message 
appears |>» prompt window •. 

" Please- inMteate. a desMnccbovx 

. With erH^r button. 11 


3. I(^u ^fc£|flpeflrs. 

Cwnor cMpw^a i\*ta> svvui £t 
verdcM of # icon. 


vl 

[A 








PASB 2flFi 










TA^K i 

MtfVINJG AM icon 

SYSTEM- 

MACINTOSH 


* 

System Folder 


fbsi4ion cursor or desired 
\tor\. 


z. Frees curd Hold naouse . 

1cor\ reverses video. 


3. HWe- cursor 4b desired 

^c>6\Hov\. "<Shos+" iwojt. fotiows 
rwexr&vdr <£ cursor. 


4. Release mouse. lotftfen, Icon 

disappears and appears in 
position <4- "gha&V. 

<feK/P> 








N\0VIKJ6 Am icon 

inte^li sp -12 


OF -2. 


tTOlV 


f> Pos'chcrn cursor ov\ (Aesl'r&tA 


2. FVe&s av\d h^4c< rftjkf r^>M5<e- 
b<4fcm. ( Menu pops up.^ 


3. ^bsrfioK ^urs^r o*\ " Wove. w 
(^wwuvvi^. ("Wove." v&rey&s 

Vfri&oOj 


4- feleaze. rfgM* mewse lc>u#ova. 

^Menu cvvH dteAjtpeair . B 0 * 

appears around iron . 

6Mrtpac< s^rsor in 4rp r^M - 6?rner.^ 











M£V 1 N<$ AM VjCM 


PfiGB 2 OF ^ 


*5. Use Cwr<or +0 " drag M lococ f» 
de&ired IcC^lbrv. 


FVess lepf or middle, mouse. 

btdtevv. (/As lotcH&vv & 

cursor c*&, n^es -te cross sH«pe. 


. C ^ bi^tfon i's 
released, icon ctfe^jojoairs on 4 
reapioenrs fo nevu location. 
Cm%or i rfeuer+K, -fb normal shape.^ 

^^toD> 








*06tc s 

a^NlN6 AM lCOV\ 



1 3. 

SYSTEM •' 

STAK 





, 4\ 

'l 


I, f&srH on cursor on c*e&\reA Icovy, 


2 - P^ess left - rnow^e bu+terv . Icon 
reverses video. 


Mj g 




Soles 

Report 


3. Ptess "OPEN" button. 


3. ^cnctu'iwfid ^ Icon disappears , 
leaving "Kcsle “ . Cursor changes 
itrb hour gloss. 




1A6K: 

0fe\JlM£ AN) I&2M 


Bftse ao=a 

5YSTEM: 

< bma. 




3 . (Continued ") fyrvfMf wirdcur' 

appears on leA- screen. 


4 Cursor reueA*. -fo rormaJ ‘skafe. 


4- (ccvifiKW*^') Ccrd&\ds> of- \co w, 
Appear fw wmcHpw. 

< EK)t>> 

V 



/rite .'Faint = 


1> Posfacm cursor cm icon 

rept&sewAivWj rtesireA iWe?rnWi* *M 


2. Pirese m^usc. butter -hvice.. 

C After- ftrsk f*&ss, 

*7&j&re£& video.') 


j Write 


eP^aint = 


2- cm Wjnwc^. £ AAer ^eaMc\ 

iCCm -furns daA: Cjray-') 


•See. Hex+- -fraw * 2 -. 


1^610 

0?£M 1N)£ AN) 1£#N) 

P&SE Q. OF £1 

SYSTEM: 

macintosh 



4 File Edit Uiew Special 



2, ccvTrw&k* C OiM iVie. of- Ipctk detaches trpm (C&a, shrinks cv*tk rvtaves 
i twavds, Centex c& &cxee *\ , -the*. g«n*/ s iz> £tze cuAh sFw^oe eA wjiwAow. 


* File Edit Piety Special 


ID 


7 items 


IDrite/Paint 


377K in disk 


□ □ <#1 

Empty folder System Fo der MocVrite 


A/»\ 

%i [g| \^i 

Font Mover Mac Advice MacPaint 




2. ( WM*w Ofpew%. ) 



<SJP> 






APPENDIX A 



WINDOW DESIGNS 
84.0816 S. Card 


(USER) (SYSTEM) 

1. Opan Window -(= Expand Icon) 


Star : 


P[ icon] 
K 'M3] 
K[OPEN ] 


Cursor arrow follows mouse. 
Icon reverse video. 

Icon turns into hole; 

Cursor «- Hourglass; 

Empty window appears 
Cursor <- Normal arrow; 
Window fills 


Mac i ntosh ; 

P [ i con ] 

K 'Ml] 

K [ M 1 ] (ra) 


Cursor arrow follows mouse. 

Icon reverse video. 

Icon turns dark gray; 

Copy of icon outline shrinks to dot as it moves to 
center ; 

Dot expands to window. 


Lisp: 


P[ 

KD 

P2 


icon] 

_ M3] 

" m E x p a n d ] 


KU[M3] 


Cursor arrow follows mouse. 

Menu pops up. 

Reverse video bar follows mouse. 
W i ndow appears . 


--or-- 


PTicon] Cursor arrow follows mouse. 

K[M2] Menu expands. 


2. Close Window ( = Shrink to Icon) 

Star : 

PfCLOSE] Cursor arrow follows mouse. 

K[M1] ’’CLOSE" box reverses video; 

Cursor ♦- Hourglass; 

T i tlebox shrinks ; 

Window disappears and icon outline fills. 


Lisp: 


P[wi ndow] 


KD 


[M3] 


KU 


P2 [Shrink] 


M3] 


Curor arrow follows mouse. 

Menu pops up. 

Reverse video bar follows mouse. 
Window disappears; 

Icon appears. 


3. Move window. 


Star : 


P[ 1 rsq] 
'‘‘Ml] 
MOVE] 


P3 

KD 

KU 


1 oc] 

[Si] 


Curor arrow follows mouse, 
lrsq box <- reverse video. 

Instruction message appears in message window; 
1 rwq box «- normal video; 

Cursor <- arrow with box; 

Cursor boxarrow follows mouse. 

Bottom of box dotted line follows mouse. 

Cursor «- hourglass; 

Bottom of box reappears in new position; 



2 


3 


Cursor ♦- arrow; 

1 rsq box <- reverse video (to indicate selected). 

Macintosh: 

? 


Lisp: 

P[window] Curor arrow follows mouse. 

K D [ M 3 ] Menu pops up. 

P2‘Move] Reverse video bar follows mouse. 

KU[M3] Cursor <- Square; 

Dotted window outl i ne appears ; 

P3[loc] Dotted line box follows movement of cursor. 

KD'Ml] Cursor *- Crosshair-. 

KU[M1] Full window appears in outine; 

Cursor Arrow. 

4. Adjust size of window. 

Star: 

Pflrsq] Curor arrow follows mouse. 

K "Ml] lrsq box «- reverse video. 

K[MOVE] Instruction message appears in message window; 

1 rwq box *- normal video; 

Cursor arrow with box; 

P3[loc] Cursor boxarrow follows mouse. 

KD'Ml] Bottom of box dotted line follows mouse. 

KU[M1 j Cursor «- hourglass; 

Bottom of box reappears in new position; 

Cursor arrow; 

lrsq box «- reverse video (to indicate selected). 


Cursor arrow follows mouse. 

Menu pops up. 

Reverse video bar follows mouse. 

Cursor «- Arrow and pointything; 

Small 'dotted rectangle outline appears; 
Dotted line box follows movement of cursor. 
Cursor <- LRCorner. 

LR Rectangle corner follows mouse. 

Blank window appears, size of rectangle 
Window fills in; 

Cursor Arrow. 

-- Alternative to last action -- 

KD[M3] Cursor «- Tongs. 

KU "Ml] Cursor «- Open tongs. 

P3'loc] Fongs follow mouse. 

KD'Ml] 

KU'M3] Cursor <- UL corner. 

P3'loc] UL corner of box follows mouse. 

KU[M1] Blank window appears, size of rectangle 

Window f i 1 1 s i n ; 

Cursor *■ Arrow. 


Macintosh: 

P[lrsq] Curor arrow follows mouse. 

KDfMl ] Dotted line appears on bottom and right side. 

P3’loc] Dotted bottom and right follow mouse. 

KU[M1 ] Bottom and right side of window fram disappears 

and reappers in new position. 


SB 


Lisp : 

Pfwi ndow] 
KD[M3 ] 

P2 ’Shape] 
KU[M3] 

P3[ 1 oc] 

KD 'Ml] 

P3 ' 1 oc] 
KU[M1] 



Layout 4: The Fisheye View - A Short Description 

By: l.cnnart Lovstrand 

Filed, in: {Phvlum}<LovstrandXThan>Layout4.tedit 

Date: 24-Jul-85 15:47:30 


The Layout 4 displays an idn tree in a fisheye view. It scales all nodes according to their distance from a 
chosen point or level of focus, keeping the proportions of each node’s box constant. In order to closer 
describe how the scaling- is performed, a piece of mathematical framework is needed. Assume a world 
concisting of a two-dimensional plane where a tree is being displayed as a collection of rectangles and 
connecting lines. Besides its "natural" vertical order ranging from root to leaf, it is also considered to have a 
horizontal order - here represented by branch numbers - ranging from a node’s leftmost branch to its 
rightmost. We are now ready to define our basic entities: 

A branch number is a non-negative integer, representing the n* leftmost branch relative to some node in the 
tree, with 0 denoting the far leftmost branch on each level. 

A node position is a sequence of branch numbers , representing a specific node in the tree. The node is found 
by walking along each of the branches relative to the root of the tree, with the empty sequence denoting 
the root of die tree. 

The level of a node position is the length of its sequence, with 0 denoting the root level. 

Finally, we need the function least upper bound ( lub ) of two nodes, which have the normal meaning of 
denoting the closest node that is "above or equal" to both of them. 

This framework lets us to continue by defining something more useful, such as the walking distance (dj from 
one node to another, informally described as the number of levels to pass in going from one of the nodes via 
the level beneith their lub to the other node. We also want the branching distance (d\), which is the absolute 
difference between the two node’s branch numbers relative to their lub: 

djia , b) = level(lub(a, b ))- 1 - leveled) + level(lub(a, b))-l- level(b ); 
d/a, b) = | branch(Iub(a, b), a) - branch(lub(a, b), b) \ 

In Layout 4, the sizes of the displayed boxes have been calculated as s v d » * sh d \ i.e. the vertical scaling factor 
(i v ) raised to the power of die walking distance (d w ) multiplied by the horizontal scaling factor (s/,) raised to 
the branching distance (dj); all measured from the focus to each of the nodes. This formula makes nodes 
appear exponentially smaller the further they are away from the focus. A small example might give a better 
idea of what is being performed: 



Figure 1 shows an "normally" formatted tree, where each node is scaled to half of it’s parents size (scaling 
factors being 5 /, =0.7, s y =0.5; the /in die figure markes die focus; nodes a and b arc for future reference). 
The view was actually produced using Layout 4 with the FOCUS property set to 0 (the root). Now we 
continue to something more spectacular in figure 2. 
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Here, the point of focus have been set to (1 1), but with no change in scaling factors. [Remember: Branch numbers 
start with zero] The walking distance between the focus and node a (path being boldfaced) is 0 (/lies on the 
level under their lub)+ 1 (a is one level further down), while the branching distance is 3 (a’s branch is three 
branches away from / s branch). This makes the ratio between the size of the node in focus and the size of 
node a be 0.5 0+1 * 0.7 3 = 0.1715. Node b's size is similarily calculated as 0.5 1+1 * 0.7 1 = 0.175, making it 
appear about same the size of node a. Using Layout 4. one may also select a complete Ivel of the tree as focus 
by setting the FFOCUS property to a single number instead of a list (mathematically by keeping the branching 
distance constant at 0). Previously this was done to produce the "normal" kind of view in figure 1; now we set 
it to a couple of levels further down the tree, giving a resulting view that in like figure 3. 



a 

Fig. 3: Fisheye Layout (Level in Focus) 


Here, the focus has been set to level 2, while still keeping the scaling factors to their original values. Usually, 
this method does not give a very good display unless it is applied to a level with a modest amount of nodes. 

It is also possible to change the view by adjusting the horizontal and vertical scaling factors. If you, for 
example, would like to «cmphasize» the path from the root to the focus while almost ignoring the 
surrounding nodes, setting sh to a high value (close to one) and sv to a low (round zero) will «accomplish» 
something like this. Following are a couple of different samples produced using Layout 4 with the differing 
parameters attached to each figure. They are all produced using the ORG. FAKE .A. CHART function of 
ORG-CHART with the parameters max node count set to 80, average branching factor set to 3, and substructure 
probability set to 0.8). 
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Layout 4, as well as the other layouts of ORG-CHART, is driven by network properties, put on the root. The 
most interesting ones are die following: 


Name 

PCT. BORDER. SPACE 

HORIZONTAL. SCALING. FACTOR 0.5 

VERTICAL . SCALING . FACTOR 

BOX. HEIGHT/BOX. WIDTH. RATIO 0.67 

SIBLING. SPACE /BOX. WIDTH. RAT 10 

FAMILY. SPACE/BOX. WIDTH. RAT 10 

FOCUS 

focus 


Dfltval Description 

0 . 1 same as Layout 3 

see above 

0.7 see above 

same as Layout 3 

0 . 5 same as Layout 3 

1 . 0 same as Layout 3 

0 the node (LISTP) or level (SMALLP) of 


«motivationcr, applikationcr?» 

«mer volym» 

«perhaps including multiple focuses» 



